Skip to content

076 GitHub copilot cost optimization - #1237

Open
Andy Huang (Whowong) wants to merge 62 commits into
microsoft:masterfrom
Whowong:xxx-GitHubCopilotCostOptimization
Open

076 GitHub copilot cost optimization#1237
Andy Huang (Whowong) wants to merge 62 commits into
microsoft:masterfrom
Whowong:xxx-GitHubCopilotCostOptimization

Conversation

@Whowong

Copy link
Copy Markdown
Contributor

This is the initial pull request for the new GitHub Copilot Cost Optimization WTH

Andy Huang (Whowong) and others added 30 commits November 29, 2023 09:27
Updated README to clarify prerequisites and challenges.
- Updated Challenge-05 to use GitHub Spec Kit with real commands
- Added proper installation steps using uv tool install
- Included full workflow: constitution → spec → plan → tasks → implement
- Updated Coach guide with troubleshooting for uv installation
- Added reflection questions and learning resources
- Add Part 1: Build app with ad-hoc prompting (no structure)
- Add Part 2: Build same app with Spec Kit workflow
- Add Part 3: Compare results between approaches
- Remove installation instructions (moved to Challenge 0 prerequisites)
- Clarify that Spec Kit is one of many spec-driven approaches
- Update Coach guide to match new structure
feat(challenge-05): Replace spec-first with Spec Kit hands-on workflow
Andy Huang (Whowong) and others added 17 commits July 1, 2026 15:56
Adds Part 3 (judge quality on an open-ended planning task) after the
car wash example, with a scoring rubric and cost tie-in. Renumbers the
compare and classify sections to Parts 4 and 5.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e students and changed the coach guidance accordingly
Removes the deterministic car wash task and promotes the open-ended
pickleball planning task into Part 2. Rewords the intro, no-retry rule,
compare/classify sections, success criteria, and reflection questions to
fit a checkable + open-ended task pair instead of easy/hard.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updated the challenge description to clarify the difference between direct prompting and spec-driven development using GitHub Copilot. Adjusted the structure and content to enhance understanding of the experiment objectives and requirements.
Removed additional challenge section regarding status field updates and comparisons.
Updated the Coach's Guide for Challenge 01 to enhance clarity and structure, including detailed instructions for comparing direct prompting with Spec-Driven Development.
@Whowong
Andy Huang (Whowong) requested a review from a team as a code owner August 31, 2026 17:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Initial content drop for the new 076 – GitHub Copilot Cost Optimization What The Hack. It adds the student challenge materials, supporting resource assets (including a small NYC starter app), and coach guides/solutions (including a model-measurement harness).

Changes:

  • Added the full student challenge write-ups (Challenges 00–06) and the hack landing README.
  • Added student “Resources” content, including the CityScout NYC mini-app and Challenge 03 model-selection worksheets.
  • Added coach guides and solution assets (including measured results + a reproducible harness).

Reviewed changes

Copilot reviewed 50 out of 53 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
xxx-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json Duplicate/placeholder Resources devcontainer under xxx- path.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge06/data/sales.csv Challenge 06 sample dataset.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/README.md Student-facing instructions for Challenge 03 resource pack.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/prompts.md Verbatim prompts + answer key guidance for model comparison.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/output-constraints.md Output-constraint cheat sheet to reduce output tokens.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-03-ModelSelection/measurements-template.md Tables/template for recording token usage and outcomes.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/restaurants.ts Restaurants route for the NYC demo app.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/neighborhoods.ts Neighborhoods route for the NYC demo app.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/routes/events.ts Events route stub for students to modify in Challenge 02.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/restaurants.ts Sample restaurants data.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/neighborhoods.ts Sample neighborhoods data.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/src/data/events.ts Sample events data.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/README.md Student README for the NYC demo app.
076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/.github/copilot-instructions.md Intentionally bloated Copilot instructions for scoping exercise.
076-GitHubCopilotCostOptimization/Student/Resources/.gitkeep Keeps Resources folder in source control.
076-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json Resources-local devcontainer configuration.
076-GitHubCopilotCostOptimization/Student/Challenge-06.md Student instructions for the Token Golf competition.
076-GitHubCopilotCostOptimization/Student/Challenge-05.md Student instructions for context window management.
076-GitHubCopilotCostOptimization/Student/Challenge-04.md Student instructions for tool overhead + cache invalidation exercise.
076-GitHubCopilotCostOptimization/Student/Challenge-03.md Student instructions for model selection + output constraints.
076-GitHubCopilotCostOptimization/Student/Challenge-02.md Student instructions for context engineering + NYC app task.
076-GitHubCopilotCostOptimization/Student/Challenge-01.md Student instructions for direct prompt vs spec-driven development.
076-GitHubCopilotCostOptimization/Student/Challenge-00.md Student prerequisites/baseline setup instructions.
076-GitHubCopilotCostOptimization/README.md Hack landing page + challenge index.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/TODO.md Author TODOs / deferred items for Challenge 03 assets.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/results.md Measured answer-key results for Challenge 03.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/README.md Coach README for Challenge 03 harness + results.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/harness.py Repro harness to measure pass rate + token usage via GitHub Models API.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/restaurants.ts Coach solution for restaurants route.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/neighborhoods.ts Coach solution for neighborhoods route.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/events.ts Coach solution for /events/free-this-week.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/restaurants.ts Coach solution data: restaurants.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/neighborhoods.ts Coach solution data: neighborhoods.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/data/events.ts Coach solution data: events.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/README.md Coach README describing the optimized Challenge 02 solution.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/skills/free-events-endpoint/SKILL.md Example coach skill for the free-events endpoint pattern.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/events.instructions.md Example scoped instruction file for events route work.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/api.instructions.md Example scoped instruction file for API routes.
076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/copilot-instructions.md Example reduced global Copilot instructions.
076-GitHubCopilotCostOptimization/Coach/Solutions/.gitkeep Keeps Solutions folder in source control.
076-GitHubCopilotCostOptimization/Coach/Solution-06.md Coach guide for Token Golf competition.
076-GitHubCopilotCostOptimization/Coach/Solution-05.md Coach guide for context window management challenge.
076-GitHubCopilotCostOptimization/Coach/Solution-04.md Coach guide for MCP tools overhead + cache invalidation challenge.
076-GitHubCopilotCostOptimization/Coach/Solution-03.md Coach guide for model selection + output constraints challenge.
076-GitHubCopilotCostOptimization/Coach/Solution-02.md Coach guide for context engineering challenge.
076-GitHubCopilotCostOptimization/Coach/Solution-01.md Coach guide for direct prompt vs SDD challenge.
076-GitHubCopilotCostOptimization/Coach/Solution-00.md Coach guide for prerequisites/baseline challenge.
076-GitHubCopilotCostOptimization/Coach/README.md Coach landing page + guide index.
076-GitHubCopilotCostOptimization/.wordlist.txt Wordlist for spelling/lint tooling.
.devcontainer/xxx-GitHubCopilotCostOptimization/devcontainer.json Duplicate/placeholder devcontainer under xxx- path.
.devcontainer/devcontainer.json Existing repo-level devcontainer definition (dotnet) present in repo.
.devcontainer/076-GitHubCopilotCostOptimization/devcontainer.json Devcontainer for the 076 hack workspace.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

- ❌ unrelated layers (frontend, security, logging, testing)

👉 Students should REDUCE this drastically.
`` No newline at end of file
Comment thread 076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/README.md Outdated
Comment on lines +9 to +14
eventsRouter.get('/free-this-week', (req, res) => {
return res.json({
count: events.length,
events
});
});

## Description

All teams will receive the same coding task with defined acceptance criteria from your coach. The task specification (found in the `/Challenge06/` folder of Resources.zip) includes functional requirements, tests that must pass, required documentation, and code quality standards.

**You must complete the Token Golf task yourself before the hack:**

1. Implement the coding task specified in `/Student/Resources/Challenge06/token-golf-task.md`
Comment on lines +21 to +25
"codespaces": {
"openFiles": [
"076-GitHubCopilotCostOptimization/Student/Resources/README.md"
]
}
Comment thread .devcontainer/xxx-GitHubCopilotCostOptimization/devcontainer.json
Comment on lines +19 to +23
"codespaces": {
"openFiles": [
"README.md"
]
}
Comment on lines +1 to +5
{
"name": "076-GitHubCopilotCostOptimization",
"image": "mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm",
"hostRequirements": {
"cpus": 2
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 50 out of 53 changed files in this pull request and generated 13 comments.

Suppressed comments (21)

Previously missed (13) — in code that hasn't changed since the last review.

076-GitHubCopilotCostOptimization/Student/Challenge-02.md:30

  • Agent Skills are discovered and loaded on demand when Copilot determines they are relevant; they do not require explicit invocation. This instruction teaches the wrong activation model and may invalidate the context-cost comparison. Describe the skill as relevance-triggered/on-demand instead.
    076-GitHubCopilotCostOptimization/Student/Challenge-02.md:118
  • This path-specific instruction guidance contradicts the supported structure already described on line 28. Scoped instructions belong under .github/instructions/ and use applyTo frontmatter; placing /api/.github/ inside the source tree will not create the intended scope.
    076-GitHubCopilotCostOptimization/Student/Challenge-03.md:34
  • This tells students to rerun anomalous results immediately after the first-shot rule forbids reruns. That changes the sample count and undermines a consistent comparison. Keep the measurement single-shot here; discuss repeated validation separately from the recorded run.
    076-GitHubCopilotCostOptimization/Coach/Solution-02.md:98
  • Copilot does not scope repository instructions by walking nested .github/copilot-instructions.md files. The reference solution correctly uses .github/instructions/*.instructions.md with applyTo frontmatter, so this troubleshooting advice sends coaches toward an unsupported structure.
GitHub Copilot walks up the directory tree looking for `.github/copilot-instructions.md` files. Ensure:
- Directory structure is correct
- File is named exactly `.github/copilot-instructions.md` (not `.github/copilot-instructions-api.md` in the api folder)

076-GitHubCopilotCostOptimization/Coach/Solution-06.md:26

  • All five challenge references are shifted: instruction scoping is Challenge 02, model selection is 03, session/tool hygiene is 04, compaction is 05, and spec-first development is 01. The incorrect mapping will make coaches point participants to the wrong prior material.

This issue also appears on line 40 of the same file.

- Global instructions scoped appropriately (Challenge 01)
- Efficient model selection (Challenge 02)
- Clean session hygiene (Challenge 03)
- Proactive compaction (Challenge 04)
- Spec-first approach (Challenge 05)

076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/.github/instructions/events.instructions.md:11

  • This scoped rule requires location, but neither the student nor coach event fixtures contain that field, and the stated endpoint acceptance criteria require only name/date/price behavior. The completed solution therefore violates its own instructions; align the required fields with the provided schema.
- Ensure each event has name, date, and location

076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-03-ModelSelection-Solutions/harness.py:116

  • The coach README says the harness defaults to n=4, and all published reference measurements use four trials, but the actual default is 50. Running the documented command without -n therefore incurs 12.5× as many API calls and can heavily rate-limit premium models.
    ap.add_argument("-n", type=int, default=50)

076-GitHubCopilotCostOptimization/Coach/Solution-02.md:21

  • Skills are selected automatically when their metadata matches the current task, not only when explicitly invoked; @workspace /new is also unrelated to loading the added Agent Skill. This coach explanation would reinforce the same incorrect activation model taught to students.
2. **GitHub Copilot Skills (conditional cost):**
   - Only load when explicitly invoked
   - Example: `@workspace /new` only loads creation templates when called

.devcontainer/devcontainer.json:8

  • This newly added default devcontainer is for hack 031 and opens /workspace/031-DevOpsWithGitHub, not the 076 materials introduced by this PR. Because it occupies the default .devcontainer/devcontainer.json location, users can launch an unrelated .NET 6 environment instead of the 076 TypeScript/Python configuration. Remove this file or replace it with the 076 configuration.
  "name" : "DevOps with GitHub WTH in Codespaces",
	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
	"image": "mcr.microsoft.com/devcontainers/dotnet:0-6.0",
	"workspaceFolder": "/workspace/031-DevOpsWithGitHub",
	"workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind,consistency=cached",

076-GitHubCopilotCostOptimization/README.md:25

  • Challenge 01 explicitly excludes automated tests and frames the exercise as an observation, not proof that SDD reduces token burn. This summary instead promises a deterministic-control result the exercise neither implements nor requires. Align the overview with the actual direct-prompt-versus-spec comparison.
- Challenge 01: **[Spec-Driven Development](Student/Challenge-01.md)**
   - Demonstrate that deterministic controls (tests, specs) reduce trial-and-error token burn

076-GitHubCopilotCostOptimization/Coach/Solution-01.md:1

  • This guide does not use Spec Kit: it deliberately uses a single manually created sdd.md with no Spec Kit commands, constitution, plan, or tasks. Calling it “with Spec Kit” contradicts the student exercise and the rest of this coach guide.
# Challenge 01 - Spec-Driven Development with Spec Kit - Coach's Guide

076-GitHubCopilotCostOptimization/Coach/Solution-05.md:15

  • These exact context-utilization thresholds are presented as deterministic behavior, yet the guide provides no measurement basis and later requires degradation after 70%. Quality degradation varies by model, task, and context composition, so a participant may run the experiment correctly and still fail the stated criterion. Label these as hypotheses/illustrative ranges and grade the recorded observation rather than a required outcome.
- 0-60% capacity: Quality remains high
- 60-70% capacity: First signs of degradation (inconsistencies)
- 70-85% capacity: Noticeable quality loss, forgotten requirements
- 85-100% capacity: Severe degradation, expensive recovery loops

076-GitHubCopilotCostOptimization/Coach/README.md:105

  • These bullets contradict the explicit statement on lines 65–67 that the hack requires no Azure resources. Remove this leftover template content so coaches are not told to prepare nonexistent Azure dependencies.

This issue also appears on line 107 of the same file.

- Azure resources that will be consumed by a student implementing the hack's challenges
- Azure permissions required by a student to complete the hack's challenges.

076-GitHubCopilotCostOptimization/Student/Challenge-06.md:17

  • Student/Resources/Challenge06 contains only data/sales.csv; the task specification and tests promised here are missing. Consequently, teams cannot know the acceptance criteria, run the required tests, or complete the capstone. Add token-golf-task.md and all referenced test/starter files to the resource folder.
    076-GitHubCopilotCostOptimization/Coach/Solution-06.md:44
  • The same off-by-one challenge mapping is repeated in the required scorecard, so teams would attribute every optimization technique to the wrong challenge.
  - Which instruction scoping strategy (Challenge 01)
  - Output constraints used (Challenge 02)
  - Tools pruned (Challenge 03)
  - Compaction points (Challenge 04)
  - Spec-first approach (Challenge 05)

076-GitHubCopilotCostOptimization/Coach/Solutions/Challenge-02-NYCApp-Solutions/src/routes/events.ts:19

  • This comparator also subtracts Date objects, which TypeScript rejects. Compare their numeric timestamps instead.
    .sort((a, b) => new Date(a.date) - new Date(b.date));

076-GitHubCopilotCostOptimization/Student/Resources/.devcontainer/devcontainer.json:21

  • This resource-level configuration also opens a nonexistent Student/Resources/README.md, so no guide appears when the container starts. Point it to an existing resource README.
    .devcontainer/076-GitHubCopilotCostOptimization/devcontainer.json:23
  • The configured startup file does not exist: Student/Resources has no README.md. Codespaces will fail to open the intended guide. Point this to an existing resource entry page, such as the CityScout README.
        "076-GitHubCopilotCostOptimization/Student/Resources/README.md"

076-GitHubCopilotCostOptimization/Coach/Solution-06.md:14

  • The documented Token Golf task does not exist in Student/Resources/Challenge06, so the coach cannot perform this mandatory pre-work or establish par. Add the complete task/test assets before making this a critical prerequisite.
**You must complete the Token Golf task yourself before the hack:**

1. Implement the coding task specified in `/Student/Resources/Challenge06/token-golf-task.md`
2. Use optimization techniques from Challenges 1-5

076-GitHubCopilotCostOptimization/Student/Resources/Challenge-02-NYCApp/.github/copilot-instructions.md:147

  • This stray two-backtick marker is unmatched and renders as unexplained literal content in the instruction file. Remove it.
    076-GitHubCopilotCostOptimization/Coach/README.md:109
  • A complete “Suggested Hack Agenda” and “Repository Contents” already appear above; this second section is untouched template guidance with sample days and duplicated headings. Remove the duplicate template block (lines 107–131) before publishing the coach guide.
## Suggested Hack Agenda (Optional)

_This section is optional. You may wish to provide an estimate of how long each challenge should take for an average squad of students to complete and/or a proposal of how many challenges a coach should structure each session for a multi-session hack event. For example:_

Comment on lines +31 to +32
1. Locate the `baseline-task.md` file in the `/Challenge00/` folder of the Resources.zip
2. Complete the coding task described using GitHub Copilot Chat in your normal working style (don't optimize yet!)
Comment on lines +66 to +70
Give **both** models this exact prompt:

```
I want to build a pickleball app. Produce an implementation plan.
First, ask me any clarifying questions you need. Then identify the
Comment on lines +58 to +62
- Use `/context` to see which tools are enabled and how much of your token budget they consume
- The VS Code Output panel (View → Output → "GitHub Copilot") shows actual tool invocations
- Each MCP tool's schema (name, description, parameters) is sent to the model on every request
- System prompt changes (like editing instruction files or toggling tools) can invalidate cached context
- Use `/usage` to see your credit breakdown after each task
Using the NYC App you worked with in Challenge 02 (in `Resources/Challenge-02-NYCApp`), implement these four features in order:

- **Feature 1 – Restaurant Filtering:** Add query-parameter support to `GET /restaurants` so callers can filter by `borough` and/or `cuisine` (e.g., `/restaurants?borough=Brooklyn&cuisine=Mexican`). Return the full list when no filters are provided.
- **Feature 2 – Neighborhood Detail with Restaurants:** Add `GET /neighborhoods/:name` that returns the matching neighborhood object plus an embedded array of restaurants in that neighborhood (a cross-file join between `neighborhoods.ts` and `restaurants.ts` data).
- **Feature 3 – Pagination (cross-cutting):** Add `?page=N&limit=N` support to all three list endpoints (`/events/free-this-week`, `/restaurants`, `/neighborhoods`). Default to page 1, limit 10. Return `{ page, limit, total, data }`.
- **Feature 4 – Input Validation & Error Handling:** Add proper `400 Bad Request` responses for invalid query parameters and `404 Not Found` for missing resources across all routes.

These four features touch multiple files, require cross-file references, and create natural breakpoints between each feature—ideal for testing context management strategies. We recommend not committing these files to make it easier to revert.
return False
if wi == -1:
return True
return di < wi
Comment on lines +22 to +24
Using the NYC App you worked with in Challenge 02 (in `Resources/Challenge-02-NYCApp`), implement these four features in order:

- **Feature 1 – Restaurant Filtering:** Add query-parameter support to `GET /restaurants` so callers can filter by `borough` and/or `cuisine` (e.g., `/restaurants?borough=Brooklyn&cuisine=Mexican`). Return the full list when no filters are provided.
Comment on lines +50 to +52
- `/compact` to summarize and continue in the same session
- `/clear` to wipe and restart with a fresh prompt
- Start a new session (`/new`) at any breakpoint you choose
]
}
},
"postCreateCommand": "curl -LsSf https://astral.sh/uv/install.sh | sh && ~/.local/bin/uv tool install specify-cli --from git+https://github.com/github/spec-kit.git",
]
}
},
"postCreateCommand": "curl -LsSf https://astral.sh/uv/install.sh | sh && ~/.local/bin/uv tool install specify-cli --from git+https://github.com/github/spec-kit.git",
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants